Repository navigation
Fix JaccardSimilarity.exactlyEqual throwing for small theta sketches - #778
MaxFreedomPollard wants to merge 1 commit into
Conversation
exactlyEqual() sized its union with ceilingPowerOf2(countA + countB) and no bounds, so ThetaSetOperationBuilder.setNominalEntries() threw SketchesArgumentException whenever the two retained counts summed to 8 or less, for example two sketches holding one item each, or to more than 2^26. Clamp the union size to [2^4, 2^26], as jaccard() has done since apache#153 and as the C++ compute_union() does for both jaccard() and exactly_equal(). Equal sketches that retain more than 2^26 entries, which needs 2^26 nominal entries, now compare as not equal instead of throwing; the javadoc notes this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leerho
left a comment
leerho
left a comment
There was a problem hiding this comment.
Thanks for tracking this down. The diagnosis is right: exactlyEqual() missed the clamp that jaccard() got for #153. I checked the other implementations, and C++ (compute_union()), Go (computeUnion()) and Rust (union_lg_k()) all clamp the union size in both functions, so this brings Java in line with them. Nothing here affects serialization or cross-language compatibility.
I ran the theta and tuple JaccardSimilarityTest classes on this branch: 21 tests, 0 failures.
Before we merge, please make two changes in theta/JaccardSimilarity.java:
-
Don't sort the union result.
exactlyEqual()callsunion.getResult(), which returns an ordered (sorted) sketch. Only the retained count and theta are used afterward, so the sort is wasted work, and it can be expensive for large sketches.jaccard()already usesunion.getResult(false, null), and C++ usesget_result(false). -
Remove the duplicated code. Both methods now repeat the same union sizing and building, and the same "identical sets" check. That duplication is how #153 got fixed in one method and missed in the other. Please factor both into private helpers, following the C++
jaccard_similarity_base, for example:// sizes the union to hold both sketches, clamped to the legal range; result is unordered private static ThetaSketch computeUnion(final ThetaSketch sketchA, final ThetaSketch sketchB) { ... } private static boolean identicalSets(final ThetaSketch sketchA, final ThetaSketch sketchB, final ThetaSketch unionAB) { ... }
Then
jaccard()andexactlyEqual()both callcomputeUnion(...)followed byidenticalSets(...).
Your new test and the existing jaccard() tests should cover the refactor as is.
org.apache.datasketches.theta.JaccardSimilarity.exactlyEqual(a, b)throws when the two sketches are small:On the same pair,
jaccard(a, b)returns[1.0, 1.0, 1.0]and the tupleJaccardSimilarity.exactlyEqualreturnstrue.Root cause
exactlyEqual()sizes its union withceilingPowerOf2(countA + countB)and passes that straight toThetaSetOperationBuilder.setNominalEntries(), which only accepts 16 to 2^26 (JaccardSimilarity.java:120). So any two non-empty sketches whose retained counts add up to 8 or less make it throw. That includes p-sampled sketches that are non-empty but retain no entries: the count is 0, so the requested size is 1. A total above 2^26 throws too.This is #153 again. That issue reported the same expression in
jaccard(). The 2017 fix (e22e016) clamped the size there to[1 << MIN_LG_NOM_LONGS, 1 << MAX_LG_NOM_LONGS]and leftexactlyEqual()unchanged. In C++,jaccard()andexactly_equal()share one clampedcompute_union()(theta/include/theta_jaccard_similarity_base.hpp).Fix
This copies the three clamping lines from
jaccard()intoexactlyEqual(). With the clamp, the result is exact for sketches configured with up to 2^25 nominal entries, because two equal sketches of that size never retain more than 2^26 entries, so the union keeps all of them. Two equal sketches that each retain more than 2^26 entries (which needs 2^26 nominal entries) now return false instead of throwing, because the union result is cut back to 2^26 entries. A javadoc note says so, modelled on the onejaccard()already has.The new
JaccardSimilarityTest.checkMinKExactlyEqualfollowscheckMinK, which was added forjaccard()with #153.Verification
Maven Central was not reachable from my environment, so I could not run
mvn test, and the TestNG suite has not been run on this change. What I did instead, with Temurin 25.0.4.1 on macOS aarch64:src/main/javawithjavac --release 25 -Xlint:all -Xdoclint:all/protected. There were no diagnostics for this class.JaccardSimilarityTestclasses against a minimal local stand-in fororg.testng.annotations.Testandorg.testng.Assert, then called every@Testmethod by reflection.I did not run the case above 2^26. It needs several GB of heap, so that part is reasoned from the code only.
🤖 Generated with Claude Code